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ABSTRACT 



Building effective school -to-work systems requires 
evaluation, usually incorporating both process and outcome evaluations. 
Increasingly, school- to-work systems are conducting self-evaluations or 
hiring evaluation teams to determine how effectively the system is designed 
and implemented and whether it is achieving its purposes for students and the 
community. Favorable evaluation results can be used for marketing the 
school- to-work system. Designing an evaluation of a school -to-work system 
requires the following steps: (1) identify stakeholders; (2) define the 

evaluation's purposes; (3) clarify goals and objectives; (4) determine 
evaluation roles and hire an evaluator; (5) write measurable evaluation 
questions; (6) choose evaluation methods; and (7) design recordkeeping 
procedures. Examples of effective practices of evaluation of school- to-work 
programs have been identified in Oregon State's Youth Transition Program and 
the Milwaukee Public School System. (Contains 10 references and an annotated 
list of 13 organizations are included in the report. A one-page fact sheet 
describes funding of the school-to-work system.) (KC) 
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Evaluation for 

School-to-Work Systems Improvement 

The foundation of continual systems improvement is evaluation, a basic management tool 
integral to the cycle of system design, implementation, and improvement. At its best, evaluation offers 
compass readings that enable the managers of a system to navigate, continually adjusting course towards 
their basic objectives. Some evaluations assess how well or how poorly systems operate {process or 
formative evaluations). Other evaluations examine how effectively systems achieve their intended 
results {outcome, summative, or impact evaluations). 



The well-designed evaluation builds into the management cycle the capacity for self-reflection 
and improvement. Just as individuals reflect upon and learn from their experiences, systems that rely on 
evaluation as a management tool incorporate the lessons of experience into subsequent planning. This 
approach views evaluation neither as an ad hoc nor as a one-time- only undertaking, but as a step in the 
management cycle. 



Building effective school-to-work systems is precisely the kind of complex systemic change that 
cannot be accomplished without evaluation. Increasingly, school-to-work systems are conducting self- 
evaluations or hiring evaluation teams. Evaluations of complex systems, such as the school-to-work 
system, usually incorporate both process and outcome evaluations. Process evaluations of school-to- 
work examine issues related to the design and implementation of the system’s model. Outcome 
evaluations look at how effectively the system is achieving its purposes, both for students and for the 
local schools, employers, and community. It enables management ultimately to assess the impact of the 
system on their students, specifically how their students perform in school and mature developmentally, 
and how their graduates fare in the worlds of work and postsecondary education. 

Marketing is a secondary but important use of the information generated by evaluation. Studies 
that demonstrate a system’s effectiveness in achieving results help justify the system and increase its 
visibility with the public and important constituencies-policy makers, employers, parents, funders, 
students, school boards, administrators, and teachers— persuading them to support the work underway, 
even leveraging additional resources. Evidence of effectiveness also has internal value in that it affirms 
for the system’s leadership and staff the importance of their work. 

Designing an evaluation of a school-to-work system requires a series of steps: 

Identify Stakeholders. The first step in designing an evaluation is to assemble an evaluation 
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team consisting of representatives of the primary stakeholders in the school-to-work system— including 
managers, instructors, partners, students, and funders— as well as the staff members with immediate 
responsibility for seeing that an evaluation is conducted. The team’s knowledge of the system 
contributes in crucial ways to the design of the evaluation and the framing of questions. Involving 
stakeholders from the beginning engages them in the evaluation process, a step likely to improve their 
understanding and their ownership of its findings, increasing their willingness to act upon them. Their 
involvement also helps evaluators gain access to people and other information sources. 

Define the Evaluation’s Purposes. The evaluation team next discusses and defines the 
evaluation’s purposes. Is it to examine process or impact or both? They identify the primary consumers 
of the information to be gathered through the assessment, and the decisions to be made on the basis of 
that information. Answers to these questions about overall purpose, as well as the system’s clarified 
goals and objectives, shape the evaluation design. 

Clarify Goals and Objectives. The next step in conducting an evaluation is to clarify goals 
and objectives. Goals are abstract, idealized statements of desired outcomes. Objectives are these goals 
restated in operational form, spelling out in detail one or more measurable criteria for success. The well- 
stated objective is specific, relevant, measurable, and attainable. Goals and objectives provide the 
standard for assessing the effectiveness of the system and its impact. 

Determine Evaluation Roles and Hire an Evaluator. Whether to hire an internal or 
external evaluator depends on several factors, including cost, evaluation expertise, commitment, 
accessibility, and bias. An internal evaluator may bring a deep commitment to and understanding of the 
system, but may also be less objective and more resistant to recommendations for change. Expertise may 
also be an issue, since few organizations have evaluation specialists on staff. 

The most complex, sophisticated evaluation studies— besides costing more in time and money- 
do not necessarily produce the most useful findings, however. And simpler evaluation methods do not 
require experts. Managers of school-to-work systems may wish to conduct some evaluations using their 
own staff and to hire outside experts for specific, more difficult evaluation tasks. 

Writ© Measurable Evaluation Questions. The evaluation team should brainstorm the 
questions to which they would like answers. The team should narrow these down by eliminating those 
that cannot be answered by available or obtainable data. The final list of questions should be cast into 
concrete, observable, measurable terms. 

Choose Evaluation Methods. The team selects appropriate evaluation methods through a 
process that considers the questions to which answers are sought, the information sought, the best 
information sources, access to those sources, reliability of sources, and costs. Data collection often can 
be incorporated into regular management processes. 

Design Record-keeping Procedures. System staff may design record-keeping procedures 
on the system’s implementation as part of its management structure. Such procedures include periodic 
benchmarks for the collection and analysis of information drawn from system records as well as focus 
groups and interviews with key staff and participants. (Records include the proposal, mission statement, 
objectives, system description, curriculum, staff development materials, staffing structure, minutes of 
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key meetings, student application and enrollment forms, and public relations and news coverage.) Staff 
should create a management information system to track the experiences of students throughout their 
tenure in the system, including numbers and categories of students, retention and completion rates, and 
changes over time in their use of the system. 



Effective Practices 

Oregon State’s Youth Transition Program has carried out comprehensive evaluations to 
document the impact of the school-to-work system on specific student outcomes and systems change. 
Data are collected from each of the system’s local sites on a variety of demographic and programmatic 
factors, over two-year follow-up periods, and analyzed in several ways. First, outcome data are analyzed 
relative to the outcomes of comparison groups, including statewide samples of students who leave 
school and students not served by the system, and a nationwide sample from the National Longitudinal 
Transition Study. Second, the evaluators examine student outcomes separately for (a) rural versus 
nonrural participants, and (b) students identified as at risk versus those not considered at risk. The 
evaluations determined that participants consistently fare better than nonparticipants in terms of wages, 
employment, and productive engagement. Rural students were found to do as well as nonrural, at risk as 
well as those not at risk. For further information, contact Dr. Michael Benz, Associate Professor, 
University of Oregon, 175 College of Education, 5260 University of Oregon, Eugene, OR 97403-5260 
★ (541)346-1408. 

The Milwaukee Public School System contracted with SRI International of Menlo 
Park, California, to conduct an evaluation of its school-to-work initiative. SRI worked in partnership 
with the school-to-work leadership throughout the 1994-95 school year, providing regular formative 
feedback, assisting with design and implementation issues. The evaluators surveyed students and 
teachers, examined documents, and conducted case studies. In August 1995, SRI released the first phase 
of the evaluation, documenting the early stages of the design and implementation in the ten participating 
schools. In this initial process evaluation, SRI found several areas of strength and progress in the 
Milwaukee school-to-work system, including capacity building and professional development of 
teachers, its emphasis on academic rigor and equity, a flexible and flat organizational hierarchy, and the 
role of liaison between educators and employers. While the evaluation is currently looking at the process 
of developing a school-to-work district initiative, SRI will provide input for a long-term evaluation of 
the school-to-work system, focusing on student outcomes. For further information, contact School to 
Work, Milwaukee Public Schools, P.O. Box 2181, Milwaukee, WI 53201-2181 ★ (414) 475-8004. 
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Worthen, B. R., and J. R. Sanders. Educational Evaluation : Alternative Approaches and Practical 
Guidelines. White Plains, NY: Longman Publishers, 1987. 



Organizations 



The mission of the Center for Human Resources (CHR) at Brandeis University is to improve the 
quality of employment preparation and education services by combining knowledge gained from scholarly 
research and practical experience in ways that help policy makers successfully address the issues of long-term 
self-sufficiency for youth and young adults. CHR has assisted thousands of policy makers and practitioners to 
create community-wide strategies linking education reform, workforce preparation, and economic development. 
60 Turner Street, Waltham, MA 02154 ★ (617) 736-3770 ★ hn4032@handsnet.org. 

The Center on Education and Work at the University of Wisconsin is a research, development, and 
technical assistance unit of the university’s school of education whose mission is to improve the linkage between 
education and work to ensure that all citizens engage in meaningful and productive careers. The Center develops 
new knowledge through research, designs and evaluates innovations, and translates research and development 
findings into practical solutions and effective policies through capacity-building technical assistance and 
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dissemination. 964 Educational Sciences, 1025 West Johnson, Madison, WI 53706 ★ (608) 263-3696 ★ 
aphelps@mail.soemadison.wisc.edu. 



The Education Development Center, Inc. (EDC) is an international nonprofit research and 
development organization whose projects span the globe, joining research with practice to meet challenges in 
education, health, technology, human rights, and the environment. EDC’s National School-to-Career Consortium 
is a collaborative of twenty-one education, industry, labor, research, state government, postsecondary education, 
and community-based organizations established to provide technical assistance to school-to-work programs and 
systems. 55 Chapel Street, Newton, MA 02158-1060 ★ (617) 969-7100 ★joycem@edc.org. 

The mission of the Far West Laboratory for Educational Research and Development (FWL) 

is to challenge and enable educational organizations and their communities to create and sustain improved 
learning and development opportunities for their children, youth, and adults. FWL conducts research-including 
evaluation, develops products and programs, provides assistance to education agencies at all levels, 
communicates the results of research and exemplary practice, and explores linkages among other agencies and 
institutions. 730 Harrison Street, San Francisco, CA 94107-1226 ★ (415) 565-3000 ★ sananda@fwl.org. 

The Institute on Education end the Economy (IEE) at Columbia University is a multi- 
disciplinary research and technical assistance center. IEE conducts research on the implications of changes in the 
economy and labor markets for all levels of education and training systems in the United States. The Institute 
also provides technical assistance and evaluation services to schools, school districts, and states involved in 
work-related education reform. Teachers College, Box 174, 525 West 120th Street, New York, NY 10027 ★ 
(212) 678-3091 ★ iee@columbia.edu. 

KRA Corporation is a small business professional and technical services organization. Its research and 
development division focuses on the application of the social and behavioral sciences to support policy making, 
planning, service delivery and management, particularly in the area of youth programs. KRA’s client divisions ' 
include research and evaluation, organizational consulting, information technology, and communications and 
management. 1010 Wayne Avenue, Suite 850, Silver Spring, MD 20910 ★ (301) 495-1591 ★ 
Infosys@KRA.com. 

The mission of the Manpower Demonstration Research Corporation (MDRC) is to develop and 
test the effectiveness of promising education, employment, and training programs designed to help economically 
disadvantaged populations become self-sufficient. MDRC’s approach is to take promising program ideas, look 
for public agencies and community organizations willing to implement a program on a trial basis, provide 
technical assistance to help these organizations, evaluate the effectiveness of the program, and actively 
disseminate the results. 3 Park Avenue, 32nd Floor, New York, NY 10016-5936 ★ (212) 532-3200 ★ 
Ivry@MDRCny.mhs.compuserve.com. 

MPR Associates Inc. is a consulting firm specializing in public sector management, planning, and 
research. MPR has conducted numerous projects involving education for work, curriculum development, 
professional development, evaluation, data systems, and program management in areas related to education and 
job training for federal, state, and local government agencies, as well as private sector clients. 2150 Shattuck 
Avenue, Suite 800, Berkeley, CA 94704 ★ (510) 849-4942 ★ mpr@netcom.com. 

The National Center for Research in Vocational Education (NCRVE) conducts research and 
development in work-related education and plays an important role in developing new models of career- focused 
education reform. NCRVE has an ongoing partnership with teams of teachers, administrators, and counselors 
from thirty schools in twenty-six major cities. 2150 Shattuck Avenue, Suite 1250, Berkeley, CA 94720-1674 ★ 
800-762-4093, 510-642-4004 ★ slf@uclink4.berkeley.edu. 
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The National Center on the Educational Quality of the Workforce is a national research and 
development center concerned with workforce and education issues, which examines the interaction of 
employers, schools, students/workers, and public policy to determine how better connections among these 
stakeholders would improve the educational quality of the workforce and bolster the nation’s competitiveness in 
a global economy. University of Pennsylvania, 4200 Pine Street, 5A, Philadelphia, PA 19104-4090 ★ (215) 898- 
4585 ★ eqw-requests@irhe.upenn.edu. 



Public/Private Ventures (P/PV) is a national nonprofit organization whose mission is to help 
improve the policies, organizational practices, and initiatives undertaken by the private and public sectors to help 
young people, especially those from poor communities, increase their capacities to lead productive lives. P/PV 
designs, tests, and studies initiatives that increase support and access to opportunity for teenagers in low-income 
communities, and provides training and technical assistance to practitioners and programs in the youth field. One 
Commerce Square, 2005 Market Street, Suite 900, Philadelphia, PA 19103 ★ (215) 557-4400 ★ 
PPVYouth@aol.com. 



The Urban Institute is a nonprofit policy research, evaluation, and technical assistance organization 
whose goal is to produce superior quality analysis and technical assistance to achieve sustainable and significant 
improvements in social and economic outcomes. Its expertise includes monitoring and evaluation systems, 
community needs assessment, local labor market analysis, development of performance standards, staff training, 
and assessment of program delivery mechanisms. 2100 M Street, NW, Washington, DC 20037-1264 ★ (202) 
857-8676 ★ blerman@ui.urban.org. 



For more information about STW Resource Bulletins, please contact: 

The National School-to-Work Learning and Information Center 

400 Virginia Avenue, SW, Room 210 / Washington, D.C. 20024 
1-800-251-7236 / Fax: 202-401-6211 
E-mail: stw-lc@ed.gov 
Internet: http://www.stw.ed.gov 
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BUILDING A COMPREHENSIVE SYSTEM, 
NOT A CATEGORICAL PROGRAM 



The School to Work Opportunities Act passed Congress 
with strong bipartisan support and was enacted in May 
1994. Across the nation, thanks in part to the leadership 
and coherence provided by the legislation, local 
communities have created programs, and states have 
passed legislation to knit programs into systems. 

The School to Work Opportunities Act is administered 
by the National School-to-Work Office under the joint 
direction of the U.S. Departments of Labor and 
Education. The Act encourages educational and career 
opportunities for all students by creating a framework 
for business and educational partnerships at the state and 
local levels. These partnerships help students make the 
vital connection between what they learn in school and 
in the workplace and prepare them for good careers and 
advanced education and training. 



Venture Capital Funding 

The school-to-work initiative puts in place a clear, 
comprehensive framework and provides venture capital 
for states and local partnerships to design systems that 
help young people make the transition from school to 
careers and lifelong learning. The initiative does not 
establish a new program to add to the myriad of existing 
education and training programs. Rather, it puts in 
place the infrastructure for a national system that’s 
based on existing models and efforts, such as career 
academies, youth apprenticeship, Tech Prep, and 
cooperative education. 



O 




In 2001, the legislation sunsets, and school-to-work 
systems will be institutionalized at the state and local 
levels. Federal support includes: 

DEVELOPMENT GRANTS. Planning grants were 
awarded to all 50 states, Puerto Rico and the District of 
Columbia to produce comprehensive plans for statewide 
systems. Seven U.S. territories have received 
development grants. 

STATE IMPLEMENTATION GRANTS. 
School-to-work implementation funds are being rolled 
out in waves. Initially, eight states were awarded grants 
with the understanding that their efforts would, in part, 
inform and improve subsequent efforts. In order to be 
approved, these statewide plans for school-to-work 
systems had to: 

■ Provide local funding to sustain the 
school-to-careers system after federal funds sunset; 

■ Coordinate activities from related federal education 
and training programs, such as the Carl D. Perkins 
Vocational and Applied Technology Education Act; 
the Elementary and Secondary Education Act; the 
Job Training Partnership Act; the Family Support 
Act; the Individuals with Disabilities Education Act; 
and the Adult Education Act; 

■ Stimulate and support school-to-work programs to 
meet the requirements of the proposal and expand 
the program throughout the state; 
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